Report on a Controlled Experiment on the Impact of Software documentation for Perfective Maintenance

Context: Writing documents is a common part of software development practice. Software engineering researchers and practitioners have expressed concerns about costs and benefits software documentation. On one hand, the role of documentation in a development process is challenged by the Agile Development Manifesto. On the other hand, researchers and practitioners need to assess whether documentation cost outweighs its benefit. Objectives: The goal of this experiment is to analyze program comprehension and documentation usage process, for the purpose of quantifying the costs and benefits of documentation and identifying comprehension strategies and behavior patterns, from the point of view of software maintainers who are new to the system to maintain, in the context of maintaining a legacy software system. Method: We designed and executed a controlled to achieve our objective. 26 student subjects were recruited from Institute of Software Chinese Academy of Sciences. The subjects were partitioned into two groups. One group had only the source code while the other had both source code and documentation available. Each subject was asked to perform a maintenance task individually on a given legacy system. With other variables controlled, we intended to examine the performance differences between these two groups. The performance aspects that we were interested in include: task duration, task completion percentage and solution quality. Results: Our findings include: (1) Software documentation delays subjects’ completion of experiment task, yet help improve the implementation quality and make it easier for in-experienced maintainers to produce high-quality codes; (2) Maintainers spent 10% of the total task time on reading documents; (3) Programming experience does not impact the benefits or costs of document that maintainers; (4) Most maintainers use As-Needed strategy to understand program while a few use a Systematic strategy with the presence of the documents. Using Systematic strategy seems to provide more benefits to maintainers. Conclusion: Our study results show that software documentation’s benefit may not be reflected by the improved quality of the code, instead of how much time it can save maintainers. Also, the limited time that subjects spent on reading documents indicated that code is often trusted more than documents. The results have two implications: (1) If a maintenance team constantly adds new members, it is wise to invest in writing system document and bridging the knowledge gap between experienced and new programmers. (2)To in-experienced new maintainers, we should encourage a Systematic approach instead of As-Needed strategy to understand the system so as to get a ‘bigpicture’ and to make consistent changes to the code.